In [9]:
reduce_darks = Reduce()
reduce_darks.files.extend(dark_files_120s)
reduce_darks.runr()
All submitted files appear valid: S20131120S0115.fits ... S20131122S0439.fits, 10 files submitted.
Ginga not installed, use other viewer, or no viewer
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['flamingos-2','gemini','browndwarf','dwarf','dragons']
Showcase how to reduce Flamingos-2 imaging data using the Gemini DRAGONS package on the Data Lab science platform using a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing flats, darks, a bad pixel mask, and science frames, and creating a single combined stacked image.
DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.
The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.
In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini South Flamingos-2 Y-band imaging data fully. This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS Flamingos-2 data reduction process; rather, it will just show one example of a standard reduction of a Flamingos-2 imaging dataset.
The data used in this notebook example is Flamingos-2 Y band imaging from the Gemini archive of the brown dwarf W0413-4750 from the Gemini South program "A Study of the 450K Transition from T to Y Dwarf, and of the 350K Y Dwarfs", PI: S. Leggett, program ID GS-2013B-Q-15.
Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.
If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:
This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.
If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:
This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.
In either case please cite the following papers:
Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445
Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411
If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:
If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}
And if you are using SPARCL, please also add \software{SPARCL} and cite:
https://doi.org/10.48550/arXiv.2401.05576
The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.
For this notebook specifically, please acknowledge:
DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L
import warnings
import glob
from gempy.adlibrary import dataselect
from gempy.utils import logutils
from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce
from astropy.io import fits
from astropy.wcs import WCS
from astropy.utils.exceptions import AstropyWarning
import matplotlib.pyplot as plt
from matplotlib.colors import Normalize
warnings.simplefilter('ignore', category=AstropyWarning)
This data comes from a Flamingos-2 program that observed a star and a distant galaxy field with dither on target for sky subtraction.
The calibrations we use in this example include:
| Observation Type | File name(s) | Purpose and Exposure (seconds) |
|---|---|---|
| Science | S20131121S0075-083 | Y-band, 120 s |
| Darks | S20131121S0369-375 | 2 s, short darks for BPM |
| Darks | S20131120S0115-120 S20131121S0010 S20131122S0012 S20131122S0438-439 |
120 s, for science data |
| Flats | S20131129S0320-323 | 20 s, Lamp On, Y-band |
| Flats | S20131126S1111-116 | 20 s, Lamp Off, Y-band |
Downloading Y-band images from the Gemini archive to the current working directory. This step only needs to be executed once.
If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.
%%bash
# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/S20131121S0075.fits
http://archive.gemini.edu/file/S20131121S0076.fits
http://archive.gemini.edu/file/S20131121S0077.fits
http://archive.gemini.edu/file/S20131121S0078.fits
http://archive.gemini.edu/file/S20131121S0079.fits
http://archive.gemini.edu/file/S20131121S0080.fits
http://archive.gemini.edu/file/S20131121S0081.fits
http://archive.gemini.edu/file/S20131121S0082.fits
http://archive.gemini.edu/file/S20131121S0083.fits
http://archive.gemini.edu/file/S20131121S0369.fits
http://archive.gemini.edu/file/S20131121S0370.fits
http://archive.gemini.edu/file/S20131121S0371.fits
http://archive.gemini.edu/file/S20131121S0372.fits
http://archive.gemini.edu/file/S20131121S0373.fits
http://archive.gemini.edu/file/S20131121S0374.fits
http://archive.gemini.edu/file/S20131121S0375.fits
http://archive.gemini.edu/file/S20131120S0115.fits
http://archive.gemini.edu/file/S20131120S0116.fits
http://archive.gemini.edu/file/S20131120S0117.fits
http://archive.gemini.edu/file/S20131120S0118.fits
http://archive.gemini.edu/file/S20131120S0119.fits
http://archive.gemini.edu/file/S20131120S0120.fits
http://archive.gemini.edu/file/S20131121S0010.fits
http://archive.gemini.edu/file/S20131122S0012.fits
http://archive.gemini.edu/file/S20131122S0438.fits
http://archive.gemini.edu/file/S20131122S0439.fits
http://archive.gemini.edu/file/S20131129S0320.fits
http://archive.gemini.edu/file/S20131129S0321.fits
http://archive.gemini.edu/file/S20131129S0322.fits
http://archive.gemini.edu/file/S20131129S0323.fits
http://archive.gemini.edu/file/S20131126S1111.fits
http://archive.gemini.edu/file/S20131126S1112.fits
http://archive.gemini.edu/file/S20131126S1113.fits
http://archive.gemini.edu/file/S20131126S1114.fits
http://archive.gemini.edu/file/S20131126S1115.fits
http://archive.gemini.edu/file/S20131126S1116.fits\
" > f2.list
%%bash
download="True"
if [ $download == "True" ]; then
wget --no-check-certificate -N -q -i f2.list
else
echo "Skipping download. To download the data set used in this notebook, set download=True."
fi
Create a list of all the FITS files in the directory
all_files = glob.glob('S2013*[0-9].fits')
all_files.sort()
# # Check header of one raw science image
# tmp = fits.open("S20131121S0075.fits")
# tmp[0].header
DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.
This tells the system where to put the calibration database. This database will keep track of the processed calibrations we will send to it.
logutils.config(file_name='f2_data_reduction.log')
caldb = cal_service.set_local_database()
caldb.init("w")
This data set contains science and calibration frames. For some programs, it could have different observed targets and exposure times depending on how you organize your raw data.
The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.
The first step is to create lists that will be used in the data reduction process. For that, we use dataselect. Please refer to the dataselect documentation for details regarding its usage.
dark_files_120s = dataselect.select_data(
all_files,['F2', 'DARK', 'RAW'],[],
dataselect.expr_parser('exposure_time==120'))
dark_files_2s = dataselect.select_data(
all_files,['F2', 'DARK', 'RAW'],[],
dataselect.expr_parser('exposure_time==2'))
list_of_flats_Y = dataselect.select_data(
all_files,['FLAT'],[],
dataselect.expr_parser('filter_name=="Y"'))
list_of_science_images = dataselect.select_data(
all_files,['F2'],[],
dataselect.expr_parser('(observation_class=="science" and filter_name=="Y")'))
reduce_darks = Reduce()
reduce_darks.files.extend(dark_files_120s)
reduce_darks.runr()
All submitted files appear valid: S20131120S0115.fits ... S20131122S0439.fits, 10 files submitted.
Ginga not installed, use other viewer, or no viewer
The Bad Pixel Mask (BPM) can be built using flat images with the lamps on and off and a set of short-exposure dark files. Here, our shortest dark files have a 2-second exposure time. Again, we use the reduce command to produce the BPMs.
It is important to note that the recipe library association is based on the nature of the first file in the input list. Since the recipe to make the BPM is located in the recipe library for flats, the first item on the list must be a flat.
For Flamingos-2, the filter wheel’s location is such that the filter choice does not interfere with the results. Here, we will use Y-band flats.
reduce_bpm = Reduce()
reduce_bpm.files.extend(list_of_flats_Y)
reduce_bpm.files.extend(dark_files_2s)
reduce_bpm.recipename = 'makeProcessedBPM'
reduce_bpm.runr()
bpm_filename = reduce_bpm.output_filenames[0]
The F2 Y-band master flat is created from a series of lamp-on and lamp-off exposures. They should all have the same exposure time. Each flavor is stacked (averaged), then the lamp-off stack is subtracted from the lamp-on stack, and the result is normalized.
reduce_flats = Reduce()
reduce_flats.files.extend(list_of_flats_Y)
reduce_flats.uparms = [('addDQ:user_bpm', bpm_filename)]
reduce_flats.runr()
This command retrieves the master dark and flat and applies them to the science data. For sky subtraction, the software analyses the sequence to establish whether this is a dither-on-target or an offset-to-sky sequence and proceeds accordingly. Finally, the sky-subtracted frames are aligned and stacked together. Sources in the frames are used for the alignment.
The final product file will have a _stack.fits suffix.
The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.
reduce_target = Reduce()
reduce_target.files.extend(list_of_science_images)
reduce_target.uparms = [('addDQ:user_bpm', bpm_filename)]
reduce_target.runr()
image_file = "S20131121S0075_stack.fits"
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()
Filename: S20131121S0075_stack.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 217 () 1 SCI 1 ImageHDU 81 (2214, 2214) float32 2 VAR 1 ImageHDU 81 (2214, 2214) float32 3 DQ 1 ImageHDU 83 (2214, 2214) int16 (rescales to uint16) 4 PROVENANCE 1 BinTableHDU 17 12R x 4C [28A, 128A, 128A, 128A] 5 PROVHISTORY 1 BinTableHDU 17 19R x 4C [128A, 427A, 28A, 28A]
data = hdu_list[1].data
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)
(2214, 2214)
fig = plt.figure(figsize = (10,10))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='bone',norm=Normalize(vmin=1, vmax=1000),origin='lower')
#These two lines will identify the brown dwarf in the stacked image.
plt.scatter(1149,1126,marker='o',facecolors='none',s=200,edgecolors='red')
plt.text(1095,1146,'W0413-4750',c='red')
#If you would like to see the full FOV, comment out the following two lines.
plt.xlim(750,1600)
plt.ylim(750,1500)
ax = plt.gca()
ax.coords['ra'].set_ticklabel_position('l')
ax.coords['dec'].set_ticklabel_position('b')
ax.coords['ra'].set_axislabel('RA')
ax.coords['dec'].set_axislabel('DEC')
plt.xlabel('Right Ascension [hh:mm:ss]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.show()
# %%bash
# cp S20131121S0075_stack.fits final.fits
# rm -r calibrations
# rm f2.list f2_data_reduction.log S2013*
# mv final.fits S20131121S0075_stack.fits